How did extinct animals move? Paleontologists are interested in figuring this out since it can tell us more about their ways of life, such as whether they were agile enough to hunt prey. It can also provide clues about how locomotion evolved; for example, when our ancestors started to walk upright. Researchers have come up with hypotheses about the movement of long-gone species by examining evidence such as fossilized bones or well-preserved footprints. Extinct animals can also be compared to similar living ones: comparing their limb length, for example, can give an idea of their speed of movement.

Deep learning models have created great opportunities for data-driven fault diagnosis but they require large amount of labeled failure data for training. In this paper, we propose to use a digital twin to support developing data-driven fault diagnosis model to reduce the amount of failure data used in the training process. The developed fault diagnosis models are also able to diagnose component-level failures based on system-level condition-monitoring data. The proposed framework is evaluated on a real-world robot system. The results showed that the deep learning model trained by digital twins is able to diagnose the locations and modes of 9 faults/failure from $4$ different motors. However, the performance of the model trained by a digital twin can still be improved, especially when the digital twin model has some discrepancy with the real system.

Mixed Reality (MR) platforms enable users to interact with three-dimensional holographic instructions during the assembly and fabrication of highly custom and parametric architectural constructions without the necessity of two-dimensional drawings. Previous MR fabrication projects have primarily relied on digital menus and custom buttons as the interface for user interaction with the MR environment. Despite this approach being widely adopted, it is limited in its ability to allow for direct human interaction with physical objects to modify fabrication instructions within the MR environment. This research integrates user interactions with physical objects through real-time gesture recognition as input to modify, update or generate new digital information enabling reciprocal stimuli between the physical and the virtual environment. Consequently, the digital environment is generative of the user's provided interaction with physical objects to allow seamless feedback in the fabrication process. This research investigates gesture recognition for feedback-based MR workflows for robotic fabrication, human assembly, and quality control in the construction of the UnLog Tower.

As collaborative robot (Cobot) adoption in many sectors grows, so does the interest in integrating digital twins in human-robot collaboration (HRC). Virtual representations of physical systems (PT) and assets, known as digital twins, can revolutionize human-robot collaboration by enabling real-time simulation, monitoring, and control. In this article, we present a review of the state-of-the-art and our perspective on the future of digital twins (DT) in human-robot collaboration. We argue that DT will be crucial in increasing the efficiency and effectiveness of these systems by presenting compelling evidence and a concise vision of the future of DT in human-robot collaboration, as well as insights into the possible advantages and challenges associated with their integration.

In this paper, we establish a task-oriented cross-system design framework to minimize the required packet rate for timely and accurate modeling of a real-world robotic arm in the Metaverse, where sensing, communication, prediction, control, and rendering are considered. To optimize a scheduling policy and prediction horizons, we design a Constraint Proximal Policy Optimization(C-PPO) algorithm by integrating domain knowledge from relevant systems into the advanced reinforcement learning algorithm, Proximal Policy Optimization(PPO). Specifically, the Jacobian matrix for analyzing the motion of the robotic arm is included in the state of the C-PPO algorithm, and the Conditional Value-at-Risk(CVaR) of the state-value function characterizing the long-term modeling error is adopted in the constraint. Besides, the policy is represented by a two-branch neural network determining the scheduling policy and the prediction horizons, respectively. To evaluate our algorithm, we build a prototype including a real-world robotic arm and its digital model in the Metaverse. The experimental results indicate that domain knowledge helps to reduce the convergence time and the required packet rate by up to 50%, and the cross-system design framework outperforms a baseline framework in terms of the required packet rate and the tail distribution of the modeling error.

A robust grip is key to successful manipulation and joining of work pieces involved in any industrial assembly process. Stability of a grip depends on geometric and physical properties of the object as well as the gripper itself. Current state-of-the-art algorithms can usually predict if a grip would fail. However, they are not able to predict the force at which the gripped object starts to slip, which is critical as the object might be subjected to external forces, e.g. when joining it with another object. This research project aims to develop a AI-based approach for a grip metric based on tactile sensor data capturing the physical interactions between gripper and object. Thus, the maximum force that can be applied to the object before it begins to slip should be predicted before manipulating the object. The RGB image of the contact surface between the object and gripper jaws obtained from GelSight tactile sensors during the initial phase of the grip should serve as a training input for the grip metric. To generate such a data set, a pull experiment is designed using a UR 5 robot. Performing these experiments in real life to populate the data set is time consuming since different object classes, geometries, material properties and surface textures need to be considered to enhance the robustness of the prediction algorithm. Hence, a simulation model of the experimental setup has been developed to both speed up and automate the data generation process. In this paper, the design of this digital twin and the accuracy of the synthetic data are presented. State-of-the-art image comparison algorithms show that the simulated RGB images of the contact surface match the experimental data. In addition, the maximum pull forces can be reproduced for different object classes and grip scenarios. As a result, the synthetically generated data can be further used to train the neural grip metric network.

As an emerging technology in the era of Industry 4.0, digital twin is gaining unprecedented attention because of its promise to further optimize process design, quality control, health monitoring, decision and policy making, and more, by comprehensively modeling the physical world as a group of interconnected digital models. In a two-part series of papers, we examine the fundamental role of different modeling techniques, twinning enabling technologies, and uncertainty quantification and optimization methods commonly used in digital twins. This first paper presents a thorough literature review of digital twin trends across many disciplines currently pursuing this area of research. Then, digital twin modeling and twinning enabling technologies are further analyzed by classifying them into two main categories: physical-to-virtual, and virtual-to-physical, based on the direction in which data flows. Finally, this paper provides perspectives on the trajectory of digital twin technology over the next decade, and introduces a few emerging areas of research which will likely be of great use in future digital twin research. In part two of this review, the role of uncertainty quantification and optimization are discussed, a battery digital twin is demonstrated, and more perspectives on the future of digital twin are shared.

The metaverse has the potential to revolutionize the next generation of the Internet by supporting highly interactive services with the help of Mixed Reality (MR) technologies; still, to provide a satisfactory experience for users, the synchronization between the physical world and its digital models is crucial. This work proposes a sampling, communication and prediction co-design framework to minimize the communication load subject to a constraint on tracking the Mean Squared Error (MSE) between a real-world device and its digital model in the metaverse. To optimize the sampling rate and the prediction horizon, we exploit expert knowledge and develop a constrained Deep Reinforcement Learning (DRL) algorithm, named Knowledge-assisted Constrained Twin-Delayed Deep Deterministic (KC-TD3) policy gradient algorithm. We validate our framework on a prototype composed of a real-world robotic arm and its digital model. Compared with existing approaches: (1) When the tracking error constraint is stringent (MSE=0.002 degrees), our policy degenerates into the policy in the sampling-communication co-design framework. (2) When the tracking error constraint is mild (MSE=0.007 degrees), our policy degenerates into the policy in the prediction-communication co-design framework. (3) Our framework achieves a better trade-off between the average MSE and the average communication load compared with a communication system without sampling and prediction. For example, the average communication load can be reduced up to 87% when the track error constraint is 0.002 degrees. (4) Our policy outperforms the benchmark with the static sampling rate and prediction horizon optimized by exhaustive search, in terms of the tail probability of the tracking error. Furthermore, with the assistance of expert knowledge, the proposed algorithm KC-TD3 achieves better convergence time, stability, and final policy performance.

Be sure to check out his talk, "Digital Twins: Not All Digital Twins are Identical," there! As we try to bridge the gap between digital and physical systems, we increasingly hear about "digital twins." Like many other concepts (e.g., Artificial Intelligence or Metaverse) the term "digital twins" can mean very different things to different people. For some, a digital twin is intimately associated with the Internet of Things (IoT) and is the digital equivalent of a sensor or a physical asset (e.g, an aircraft engine). It allows them to experiment with the digital version that they may not be able to do with the physical system.

Digitalisation is a strategic priority for public healthcare policy: major innovations in medicine are expected from the development of IT tools and the generalisation of digital models, which will contribute to improving the efficiency of healthcare systems and services, writes Inria in this press release.. In order to guide the digital transition of this sector in the Lyon region, Inria and the Hospices Civils de Lyon (HCL), the second largest hospital centre in France after AP-HP (Paris), are pooling their expertise to create a centre for the development of artificial intelligence and a joint project team dedicated to digital models for neuroscience. On July 2021, the two bodies signed a "memorandum of understanding", prior to concluding a framework agreement for tripartite collaboration (Inria / HCL / Claude Bernard–Lyon 1 University), thus officialising the details of this shared ambition. "The signing of a large-scale partnership between Lyon Public Hospitals and Inria testifies to the firm positioning of the University Hospital and its academic partners with regard to digital healthcare", says Raymond Le Moign, Managing Director of Hospices Civils de Lyon. "For the last ten years, the application of digital science in the healthcare sector has been a key focal point for Inria. Almost a third of its teams lead research which opens opportunities in this field", according to Hugues Berry, senior researcher in molecular neuroscience and Deputy Scientific Director at Inria.